Error diagnosis and classification of errors in two Hebrew state-of-the-art automatic speech recognition systems

In this research we diagnose two commercial automatic speech recognizers (ASRs) on a corpus of academic lectures in Hebrew. Our goal is not only to measure the engines' performance but to find out if current Hebrew ASRs' transcription can be a reasonable replacement to human transcription, or at least a significant bootstrapping for a manual post-processing of the automatic output. We performed a word error rate (WER) diagnosis and a linguistic error classification on two automatic transcriptions – Nuance's and Google's, and compared it to a real-time (RT) stenographer's records, as well as to an exact transcription that reflects excatly the speaker's speech. Results show that the ASRs‘ WER is caused by massive substitutions, while the RT transcription's errors were caused mainly due to deletions. This research provides an opportunity to explore cost/benefit aspects of automatic vs. manual audio transcriptions.